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THE EFFECTS OF CREW RESOURCE MANAGEMENT (CRM) TRAEVING 

IN AIRLINE MAINTENANCE: 

RESULTS FOLLOWING THREE YEAR’S EXPERIENCE 
Report of Work Completed through 12-31-94 


SUMMARY 


This report describes three year's evaluation of the effects of one airline's Crew 
Resource Management (CRM) training operation for maintenance. This evaluation 
focuses on the post-training attitudes of maintenance managers' and technical support 
professionals, their reported behaviors, and the safety, efBciency and dependable 
maintenance performance of their units. The results reveal a strong positive effect of the 
training. 


The overall program represents the use of CRM training as a long-term 
commitment to improving performance through effective communication at all levels in 
airline maintenance operations. The initial findings described in our previous progress 
reports are reinforced and elaborated here. The current results benefit from the entire 
pre-post training survey , which now represents total attendance of all managers and staff 
professionals. Additionally there are now full results fi"om the two-month, six-month, and 
12-month follow-up questionnaires, together with as many as 33 months of post-training 
performance data, using several indicators. 


In this present report, we examine participants’ attitudes, their reported behaviors 
following the training, the performance of their work units, and the relationships among 
these variables. Attitudes include those measured immediately before and after the 
training as well as attitudes of the participants months after their training. . Performance 
includes measures, by work units, of on-time flight departures, on-schedule maintenance 
releases, occupational and aircraft safety, and efficient labor costs. We report changes in 
these performance measures following training, as well their relationships with the training 
participants’ attitudes. 


r 


Highlights of results from this training program include increased safety and 
improved costs associated with positive attitudes about the use of more assertive 
communication, and the improved management of stress. Improved on-time performance 
is also related to those improved attitudes, as well as favorable attitudes about 
participative management. ^More,specific results^are as follows:.^ ^ 

y'. ' ■ 


1) We note positive trends in a number of the company's overall maintenance 
performance indicators for the months after the onset of training (variously 27 to 34 
months, depending on the measure), compared with the months before (varying from 5 to 
17 by measure). Measures in the Safety Performance category improve in the 32 months 
measured after the training. In addition, results in the Dependability Performance category 
continue to improve in the 27 months measured after the onset of training. The available 
measure for Efficiency also shows generally effective performance during the first 18 
months following training, but then performance slips for the remaining 16 months 
measured. 


2) Comparisons of managers' attitudes immediately after their training, with their 
pre-training attitudes, show significant improvement for most of those expected. 
Improvement is noted in attitudes about "willingness to share command responsibility," 
"usefulness of communication & coordination," and "recognition that stressors effect 
management decision making." The same attitudes of these managers two-, sbc-, and 12- 
months later, reveal that these favorable post-training attitudes remain at those high levels 
in the months after the training. The fourth attitude scale measured, “willingness to voice 
disagreement” (a measure of assertiveness), although showing no significant change 
immediately following training, improved significantly above the pre-training levels two 
months after training and it remained at that higher level sbc and 12 months afterwards. 
The influence of the training on all the expected participant attitudes is thus a stable and 
robust change, and not merely a brief "honeymoon effect. 


3) Other analyses of the data look at the relationships between pre-, and post- 
training attitudes and maintenance performance over several years. In this "time-lagged" 
correlation analysis we tested the effects for performance before the onset of the training 
program, before the subsequent surveys, and for effects on performance months after the 
training and/or follow-up surveys began. 


We found a pattern of significant relationships between the participants’ pre- 
training and post-training attitudes and performance. Attitudes about skills and behaviors 
specifically taught in the CRM training were more predictive of subsequent maintenance 
performance. Attitudes, immediately following training, about the value of 
communication & coordination, as well as about the “value of stress management,” were 
found to be related to on-time scheduled maintenance performance both prior to and after 
the training began. 
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The strongest effects of CRM training appear on respondent attitudes two months 
after their training. Examination of that two-month follow up survey reveals that attitudes 
about sharing command responsibility, and about assertiveness (both of these are "active" 
beliefs emphasized in the training, and which require behavioral support to reinforce 
them), are related to subsequent on-time flight departures, and occupational safety. 
Attitude about sharing responsibility is also related to many months of timely maintenance 
work both before and after the training began. Overall, the correlations between 
performance and the attitudes measured in this two-month follow-up survey are 
remarkably strong and indicate that lessons learned from the CRM training have 
consolidated by two months afterward, and in "active" ways. 


The six-month follow-up results show a retreat from the effects in the 2-month 
survey. The 6-month results reveal that only prior maintenance performance is related to 
surveyed_opinions about delegation, communication and stress management. 


By the 12-month follow-up survey, the CRM training effects have returned to 
mirror aspects of the post and 2-month surveys. Results of this later survey suggest that 
the lapse at 6 months probably represents a consolidation of the training effects rather than 
their elimination. Training participants continued to develop the skills they learned in 
CRM and used them to good effect. Attitudes about the value of assertiveness one year 
after CRM training were related to many months of occupational safety both before and 
after that survey was taken. Attitudes, a year after training, toward sharing command 
decisions were (reminiscent of the 2 month results) also related to many months of 
occupational safety. Finally, 12-month attitudes about managing stress effects were 
associated with improved performance in the very stressful performance areas of meeting 
ambitious goals for cost saving, and for timely maintenance. These findings strongly 
suggest that lessons of stress management, learned during training more than a year 
earlier, had also developed and matured for the training participants. 


4 


THE EFFECTS OF CREW RESOURCE MANAGEMENT (CRM) TRAINING 

IN AIRLINE MAINTENANCE: 

RESULTS FOLLOWING THREE YEAR’S EXPERIENCE 


J.C. Taylor Ph.D. & M.M. Robertson Ph.D. 
Institute of Safety and Systems Management 
The University of Southern California 
Los Angelest December 1994 


THE STUDY 


Background 

The CRM program reported here is first-time training in several team-related 
concepts, including communication skills, self-knowledge, situational awareness, and 
assertiveness skills for maintenance management and support staff personnel in one large 
U.S. airline (hereinafter called "the company"). Maintenance (or "Technical Operations") 
in this company includes engineering, quality assurance, technical planning, systems & 
procedures, contracts administration and purchasing, as well as the more direct 
maintenance functions of line- and base-maintenance, inspection, shops and material 
services. The company succeeded in training all of its maintenance directors, managers, 
supervisors and assistant supervisors, engineers, planners, coordinators, and schedulers, as 
well as a large part of its inspection staff According to company records, 2,199 people in 
total completed the CRM for Maintenance course between June 1991 and March 1994. 

CRM training in airline maintenance operations was highly unusual when 
introduced into the present organization. This exceptional example has been intensively 
studied over a three year period. The first papers (c/, Taylor, 1991a, 1991b) reported the 
methodological and measurement characteristics of the attitude and performance 
indicators we used. Subsequent reports have been limited to a relatively small sample of 
participants, and a small number of months of maintenance performance data subsequent 
to the training. A summary report of the first six months experience (prepared, by Taylor, 
Bettencourt & Robertson, at the University of Southern California in 1992) was published 
in the FAA Office of Aviation Medicine's second annual human factors R&D progress 
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report (Galaxy, 1993); and a subsequent report on the first full year’s experience (also 
prepared at the University of Southern California in 1993) has been included in the FAA 
Office of Aviation Medicine's third progress report (Galaxy, 1994). Several shorter papers 
dealing with selected topics fi'om this study have also been published (Stelly & Taylor, 
1992; Taylor, Robertson, Peck, & Stelly, 1993; Robertson, Taylor, Stelly, & Wagner, In 
Press). The present report concludes the evaluation of the company’s initial program, by 
documenting the mutual effects between training-related attitudes and behaviors and work 
unit performance, as well as the persistence and stability of the attitude and behavioral 
changes following the training. 

The effectiveness of this training, as measured by its ongoing evaluation, can help 
direct the airline industry's maintenance human resources practices in the future and guide 
the development of future AT A and FAA training policies and regulations. 

The analyses reported below assess the relationships among respondents' training- 
induced attitudes about a variety of management and organizational items and 
maintenance unit performance. 


The Purpose of the Program and of the Course. 

The program's champion was the company’s Senior Vice President for Technical 
Operations. He announced that his aim for the joint training and evaluation program was 
to improve human resource (HR) management by using science-based tools and 
techniques for evaluating the training outcomes and using those results for continuously 
improving the program's effectiveness (Fotos, 1991). 

This particular training program originally began with advice and assistance fi'om 
the company's flight operations training group who had nearly a decade's experience with 
their own CRM program. After that initial assistance, the company’s flight operations and 
technical operations (i.e., maintenance) CRM programs were directed and administered 
separately. During the first three years of that maintenance CRM program it continued to 
be managed and administered by maintenance personnel, and the trainers were mainly 
maintenance people too. 

The purpose of the training, as reiterated by trainers on the first day of each 
training session, was "To equip ail maintenance personnel (management first) with the 
skill to use all resources to improve safety and efficiency." 

Course objectives. The objectives (the more specific goals of the training) were 
also clearly stated during the trainers' introductory remarks; 


1) Diagnose organizational "norms" and their effect on safety. 

2) Promote assertive behavior. 
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3) Understand individual leadership styles. 

4) Understand and manage stress. 

5) Enhance rational problem solving and decision making skills. 

6) Enhance interpersonal skills 

The course was designed for the objectives. The aims and objectives of the 
training were achieved by following a course syllabus containing 12 modules (Appendix A 
contains the syllabus). 

The first phase of training completed. By the conclusion of this phase of the CRM 
training intervention, in February, 1994, nearly 2,200 participants had attended the course. 
Called “the first phase” of training, it included all Technical Operations management and 
staff professionals, and it is the subject of this present report. During early 1994, the usual 
number of new hires or recent promotions to management ranks continued to attend the 
CRM course and those sessions were also attended, for a few months, by Aircraft 
Inspectors, as the beginning of a second phase which would include all mechanics and 
inspectors in the training. The present analysis uses data fi’om the entire sample of 
maintenance management, salaried professional staff, and inspectors (n=2,199) who 
participated in the “first phase” training course between June 1991 and March 1994. 

Individual respondents as the focus of analyses. To explore some of the effects of 
the training on all individuals, the data from all training participants will be used. The 
remaining (and majority of those) analyses, however, examine the attitudes of respondents 
who (through the use of their confidential code numbers) can be matched between the pre- 
training survey and the later ones. 

Maintenance work units as the focus of analyses. The maintenance performance 
data (classified into categories of "safety," "dependability," and "efficiency") were 
measured by work units, not by individual respondents. The analyses described in this 
report illustrate the effect of changes in respondent attitudes upon the maintenance 
performance of their work-units. For managers these are the units they lead, and for staff 
professionals, maintenance foremen, and inspectors these units are the stations and 
locations to which they belong. In order to accomplish that examination of attitudes 
correlated with performance, the individual respondent’s attitude data were combined into 
averages for the appropriate units. 


The "Crew Resources MgtyTech Operations Questionnaire" (CRIVI/TOQ) 

Prior experience in measurement of attitudes related to CRM training. The 
Cockpit Management Attitudes Questionnaire (CMAQ) has long been a recognized 
measure for assessing flight crew attitudes. It is useful as a training, evaluation and 
research tool (c/. Helmreich, Foushee, Benson, & Russini, 1986). The 1990 CMAQ 
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questionnaire contained 25 items measuring attitudes that are either conceptuaUy or 
empirically related to CRM. Taggart (1990) revised the CMAQ for use in a maintenance 
department, and reported positive initial results following CRM training conducted for 
maintenance managers in late 1989. 

Two previous studies have explored the CMAQ instrument for a consistent 
internal structure using the Factor Analysis technique (Gregorich, Helmreich, & Wilhelm, 
1990; Sherman, 1992). In these two studies, using samples of flight crews and air trafiSc 
controllers, the authors showed that the relationships among the 25 CMAQ items 
clustered into the following four constellations of attitudes: 

1) Sharing Command Responsibility, 

2) Value of Communication & Coordination, 

3) Recognizing and Managing Stressor Effects, 

4) Avoidance of Interpersonal Conflict 

Gregorich, et al. (1990) eventually reduced their set from four to three composites 
by dropping "Avoidance of Interpersonal Conflict." Sherman, on the other han4 found 
that fourth factor to be much more robust in his sample of Air Traffic Controllers, and he 
titled it "Advocacy and Assertiveness." Those authors combined the individual CMAQ 
items into three or four composite index scales, respectively, to obtain more stable 
indicators of underlying concepts. Such indices permit a more detailed assessment of the 
separate but related attitudes than a single total score for the entire questionnaire, but they 
also provide more accurate and reliable results than are available from each of the 
individual questionnaire items alone. 

Measurement of attimdes in the present study. The "Crew Resources 
Management/Technical Operations Questionnaire" (CRM/TOQ) developed for the present 
study is a modified version of Taggart's revised CMAQ, In choosing to draw on the 
CMAQ to obtain a "good" measure of the complex concepts of CRM training, the - 
company and the researchers wished to start with a survey, already proven valid as an 
measure of CRM training in the airline industry. The two parties also wanted more 
maintenance related measures than the CMAQ provided — in particular they wished to 
measure opinions, important in maintenance, but which were not expected to change 
following the training; as well as rather more specific expectations for future use, as well 
as reports of how the training was subsequently used. The CRM/TOQ followed the 
successful format set by the CMAQ, in that it is a short questionnaire, yet having enough 
items to provide convergence to a smaller, easily used, set of concepts. 

The CRM/TOQ contains 26 multiple response items. The company's modifications 
of the CMAQ involved removing five questions and adding six others. The five questions 
were removed because they either lacked predictive validity as rqjorted by earlier flight 
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crew studies (Helmreich, et al., 1986) or, in the company’s opinion, lacked relevance to 
maintenance. 

A confirmatory Factor Analysis was undertaken for the data obtained with the 
CRM/TOQ (Taylor, 1991a). Results for the items drawn fi'om the revised CMAQ were 
similar to those of Gregorich, et al. (1990), and subsequently with Sherman (1992). As in 
Sherman’s study, the CRM/TOQ's fourth composite was statistically strong, and was 
therefore retained as the reflected index "Willingness to Voice Disagreement." 

Six questions were also added to the CRM/TOQ, based on items intended to 
measure respondents' perceptions of behaviors dealing with setting and attainment of work 
goals (Geirland & Cotter, 1990). These sbc individual items were considered important to 
add because the work of maintenance managers differs fi'om that of Flight Operations’ 
officers (as managers) in the typically longer time required for Technical Operation's goal 
attainment and the relatively greater ambiguity of those goals. In addition, because goal 
setting and attainment were not covered in the CRM training such items could act as 
"control questions" about events in the respondents’ work lives which were not predicted 
to change in a consistent or positive way following the training. These questionnaire items 
were separately tested by Factor Analysis and results suggested a two factor structure: 
"Goal sharing in one’s own group," and "Goal sharing with other groups." 

Table 1 displays the six scales calculated from the CRM/TOQ and which are used 
to test the effects of the CRM training on individual participants. Table 1 presents those 
six scales and the numbers of the individual item used to calculate each one. The 
individual item numbers correspond to the numbering of the sample questionnaire located 
in Appendix B. 


TABLE 1 


Scales Used to Test Individual Effects of CRM Training ^ 

SCALE 

CONSTITUENT ITEMS (See Appendix B) 

Scales Sensitive to CRM 

Sharing Command Responsibility 

6, 8. 11, 13, 19 (reflected) 

Communication & Coordination 

5, 12, 14, 16, 17 

Managing Stress 

9, 20 

Assertiveness 


“Control Scales" 

Sharing Goals in own Group 

21,22,23,24 

Sharing Goals with other Groups 

25,26 
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Measurement Characteristics of the CRM/TOQ. Both reliability and validity of the 

separate items and the composite scales were tested, and they demonstrated good 

measurement qualities (Taylor, 1991a). 

Four versions of the CRM/TOO. There are four versions of the CRM/TOQ 
questionnaire which were used in various phases of this project. 

1. A "Baseline Questionnaire" was mailed to all 1,800 maintenance managers, 

supervisors, and assistant supervisors in the company in May 1991, before the 
training was announced. The results of this baseline assessment of management 
attitudes were used in the Factor Analysis described above and were also used to 
establish the reliability and validity of the CRM/TOQ (Taylor, 1991a). The 
Baseline data were also compared with the subsequent “Pre-training” survey and 
the comparable questions’ mean scores were found to be virtually indistinguishable 
from one another (Taylor, Bettencourt & Robertson [Galaxy], 1993) The baseline 
survey played no further role in the present study. The first 26 multiple choice 
questions were the same for all subsequent versions of the CRM/TOQ. 

2. A "pre-training" questionnaire was completed by all participants immediately before 

each workshop began. Respondents were asked to choose a private identification 
(I.D.) or code number and to write it on their questionnaire. It was explained that 
this number would allow their responses to be compared with their responses on 
subsequent, follow up questionnaires, but without identifying them. These pre- 
training attitudes were subsequently compared with attitudes immediately after the 
training, as well as with attitudes measured months later. 

3. A "post-training" questionnaire was completed by participants at each workshop's 

conclusion. Respondents, again, wrote their I.D. code on the questionnaire; and 
were asked to note that number in their course workbook to help them remember 
it for later "follow-up" questionnaires. The training facilitators collected and 
mailed the completed pre-training and post-training questionnaires to the 
University for processing. Data from this post-training version of the CRM/TOQ 
were compared with the pre-training questionnaire data. As described further 
below, three attitude scales had statistically higher mean scores immediately 
following training. The fourth attitude, “assertiveness,” was not statistically higher 
immediately after training than before. The two opinion scales about goal sharing 
were not statistically different before and after the training. 

4. "Follow-up" questionnaires were individually mailed by the company to all past 

participants two, six, and twelve months following their CRM training. 
Respondents were again asked to include their private I.D. code on their 
questionnaires, so that those follow-up surveys could be matched with the earlier 
questionnaires. They were provided an pre-addressed and stamped envelope with 
which to return the questionnaires to the University for processing. Although the 
2, 6, and 12 month follow-up surveys were all identical in form, they measured the 
respondents' thoughts, assessments, and attitudes over increasingly lengthy periods 
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from the training (an example of the CRM/TOQ Follow-up questionnaire is 
included in Appendix B). 


Questionnaire return rates. Table 2 presents the return rates for the five surveys 
reported here. The total number of surveys returned and the percentage returned are 
shown for each survey. 




Sample Size And Response Ratios: Five Surveys 
CRM/TOQ Questionnaires Received by October; 1994 (2;199) total participants): 


SURVEY 

TOTAL RETURN 

RETURN RATE 

Pre-training Survey 

2,056 

93% 

Post-training Survey 

2,053 

93% 

2-month Follow-up Survey 

810 

37% 

6-month Follow-up Survey 

722 

33% 

12-month Follow-up Survey 

540 

25% 


The high return rate for the pre and post-training surveys result firom participants 
being asked to complete the questionn^es while they were present in the training 
sessions. Lower return rates for the subsequent follow-up surveys is the normal result of 
surveys conducted by mail, but the 25%-37% return rates displayed in Table 2 are lower 
than desired. During the first year of the CRM program, we reported that the follow-up 
survey return rates ranged from 40%-45% (Taylor, Bettencourt & Robertson [Galaxy], 
1993). During the intervening two years, although maintenance management continued to 
fully support their CRM program, the company experienced considerable turbulence in its 
markets and it reacted with several waves of staff reductions and station changes 
(including closures in several cases). These changes introduced consequent personnel 
moves which made it extremely difficult for the CRM course administrators to 
subsequently locate course participants to send them surveys. The lower follow-up survey 
return rates for the entire three years reflect this administrative difficulty — using out of 
date or incorrect address lists caused many questionnaires not to be sent to their intended 
recipients. Although lower morale due to layoffs and station closures can be hypothesized 
to contribute to a lower return rate, the stable attitudes measured in the considerable 
number of follow-up questionnaires received do not reflect it. 
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Testing the effects of missing data on scale scores. In any survey instrument like 
the CRM/TOQ, individual respondents will occasionally make a mistake or omit an 
answer to a specific question. Such errors and omissions result in “missing data” 

(unusable answer, or no answer given). When individual items with missing data must be 
used with others with valid responses to form scales, such as we have done for the present 
study, the research investigator faces the choice of dealing with missing data on individual 
questions in three ways. The investigator 1) can eliminate the scale in question for that 
respondent entirely, or 2) can calculate the scale score without the individual item 
included, or 3) can substitute the population mean score for that item to replace that 
respondent’s “missing data.” Each approach has advantages. The first and second 
alternatives use only actual data, but the former reduces the sample size, and the latter 
biases the scale in favor of the remaining items. The third alternative maintains the sample 
size with a minimum distortion of the resulting score for the total sample. 

We compared both the first and the third alternatives and tested the degree of possible 
distortion introduced by replacing missing responses in individual items with the sample mean 
before calculating the scale score. There are four attitude scales, and two “control scales” (c^. 
Table 1), measured for the two pre and post-training administrations of the CRM/TOQ 
questionnaire, for a total of 12 modified scale mean scores. The increase in total sample “n” 
using this modification was between 50 and 255, depending on the particular scale in question. 
As noted above, the total number of questionnaires retumal was, 2,056 for pre-training, and 
2,053 for post-training samples (cf , Table 2). The smallest “n’s” using the unmodified scales 
were 1,801 and 1,804 for the pre-training “goal sharing” scales. After replacing the missing 
data for every individual item used to construct a scale the obtained “n” for every scale in the 
pretest and posttest was returned to 2,056 and 2,053 respectively. 

Mean values for the 12 scales, modified to correct for missing data, were compared 
with the 12 unmodified ones. Eleven of the 12 scales so modified were unchanged fi’om the 
original (unmodified) data set. Only one of the “control scales,” 'Ivty unit shares goals with 
other work units in Technical Operations, " was considerably higher in its modified form for the 
pre-test sample (0.30 scale point higher on a five-point scale). The modified post-test score for 
this scale was undifferentiated fi'om the unmodified version. For 1 1 of the 12 items tested, 
using the population mean score to replace missing data was a net benefit — it increased the 
sample size without affecting the scores themselves. The question of the outlier, however, still 
remains. 

Why should this one scale behave so differently from the rest? The answer is that a 
number of administrative and management people reported very positive contact with other 
groups and, coincidentally, they also did not answer all the questions on that topic. The 
original data collected during the first few months of the program (June-Aug., 1991) showed a 
tendency to over-evaluate the items compri^g this scale {cf., Taylor 1991a). A 
disproportionate number of respondents during those first few months of training wa^ soiior 
executives and managers. These respondents, as a group, probably feel more certain in their 
views, and thus would be expected to have a lower missing data rate, and would also be more 
likely to have an answer to questions about “other groups” in Technical Operations. After that 
initial few months, the pre-training mean scores for that scale, settled to a lower level which 
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remained unchanged from the post-training scores for the remainder of the nearly three years 
we measured. The individual questions dealing with goal sharing with other groups had a 
higher missing data rate (were answered less often) than all other multiple response questions. 
As time passed, an increasing number of respondents did not answer one or both of the 
questions asking about the topic. Later respondents probably felt less certain about this 
concept and those source behaviors. In the final sample there are 252 (roughly 12% of the 
population) pre-training subjects who did not answer one or both of the questions comprising 
the scale, "My unit shares goals with other work units in Technical Operations." When we 
examined the differences across the various departments in Tech Operations, we found that 
pre-training modified scale scores for the mechanical trades departments (line, base, and shop 
maintenance) were largely unchanged from their original data. Howeva-, personnel in the 
other Technical Operations departments (many of them higher managers) showed marked 
increases in their scores when their pre-training data set was modified, which means that tl» 
one item in the scale that they did answer, they answered very positively. Adding the sample 
aver^es to these very high scores appreciably raised the total scores for this scale. The &ct 
that his group would both answer positively and skip questions on this topic is either a 
coincidence or an unknown measurement artifact affecting that one scale. Modifying all scales 
by replacing missing data with population means was used in all analyses in the present report. 

Testing CRM/TQO questionnaire sensitivity. The quantitative questions in the 
CRM/TOQ were designed to obtain four independent attitude scales (c/C, Table 1) and sevaal 
individual opinion items, all specifically sensitive to the CRM training curriculum. There were 
also two “control” scales, dealing with perception of goal setting and attainment, which were 
specifically intended not to be sensitive to the training. As noted in the preceding section, one 
of these control scales was the only one of the six which was affected by the scale modification 
procedure to correct for missing data. Table 3 presents additional evidence for the differences 
between the CRM attitude scales and the control scales. Table 3 is a intercorrelation matrix 
between the four attitude and the two control scales from the pre-training survey and those sbc 
scales from the post-training survey. The cell entries are Pearson Product Moment Correlation 
coefficients (“r”) and those in italics are statistically significant at the .05 confidence level or 
higher. Boldface coefficients are those above r=.49, representing unusually high correlations, - 
- those which account for at least 25% of the Joint variance of the two correlated scales. That 
would mean that about 25% of a given attitude scale’s fluctuation was explained by the 
fluctuation of the other scale in correlation with it. 


TABLES 

Intercorrelations among: Pre-training and Post-traihing: Survey Scales 



Pre-Training: 

Delegating Command Responsibility 

Pre-Training: 

Communication & Cooperation 

Pre-Training: 
Managing Stress Effects 

Pre-Training: 

Assertiveness 
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Pre-Training Control: 
Goal Sharing with Others 

Posl-Training: 

Delegating Command Responsibility 

Post-Training: 

Communication & Cooperation 
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Managing Stress Effects 
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Post-Training Control: 
Goal Sharing with Others 

Pre-Training: 

Del. Command Resp. 

- 












Pre-Training: 
Communic & Coord. 

.31 

- 











Pre-Training: 

Mnq. Stress Effects 

.50 

.39 

— 











.28 

-.01 

.10 

— 









Pre-Training Control Item: 
Goal Sharing w/Gp. 

.01 

.12 

-.04 

.01 

— 








Pre-Training Control Item: 
Goal Shr w/Others 

.05 

.08 


-.01 

.74 

Bi 







Post-Training: 

Del. Command Resp. 

.52 

0.0 

.22 

.26 

0.0 

.02 

m 






Post-Training: 
Communic & Coord. 

-.03 

.38 


-.02 

.13 

.08 

0.0 

Bj 





Post-Training: 

Mng. Stress Effects 

.23 

.04 

.43 

.11 

0.0 

0.0 

.35 

.08 





Post-Training: 

Assertiveness 

.18 

-.02 

.06 

.47 

.02 

0.0 

.28 

-.04 

.11 

- 




-.14 

.11 

-.09 

-.06 

.44 

.16 

-.12 

.21 

-.10 

-.03 

- 



-.15 

.03 

■g 

-.09 

m 

m 

-.18 

.11 

-.12 

-.09 

.47 


1 n a 1,822 { 


The pattern of intercorrelations in Table 3 shows that the best predictor of later 
attitudes are the same attitudes at the earlier time period. The pattern also shows the two 
control scales are more related to themselves and to one another than they are related to the 
CRM attitude scales. These results demonstrate that the four CRM attitudes are largely 
independent of one another (a desirable condition), and that they are also little related of the 
two control scales. If anything, there is a slight tendency for the CRM scales to be inversely (if 
only modestiy) related to the control scales. That means there is a slight but believable, 
tendency for scale scores on the goal sharing scales to decrease as scores on the CRM attitudes 
increase. The patterns of intercorrelations found in Table 3 were replicated in intercorrelations 
between the Post-training survey and the 2-month, 6-month, and 12-month Follow-up surveys. 
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RESULTS SECTION 1: 




( 


COMPARISON OF ATTITUDES OVER TIME 

All available survey responses are used for these initial comparisons. 

Attitude change from pre-training to post-training surveys. Results using all 
respondents who completed the CRM/TOQ immediately before and immediately after 
training demonstrated that many of the intended effects on participants’ attitudes were 
achieved. Figure 1 presents the pre, post, 2, 6, and 12 month comparisons for the four 
attitude scales using all responses. 



The attitude index scores for "Sharing Command Responsibility," "Communication & 
Coordination," and "Managing Stressor Effects," all increased immediately following 
training. A Multiple Analysis of Variance (MANOVA) “F’ was used to test the 
differences among same-scale values for the pre- and post-training surveys and established 
that the pre-post differences noted were statistically significant statistic (F=6.77, df = 

2156, 4; p< 000). "Willingness to Voice Disagreement," a measure of assertiveness and a 
central topic of the training, did not increase significantly immediately after the training. 

Attitude change in follow-up surveys. Once again the MANOVA “F” statistic was 
used to test the differences among the same-scale values for the four surveys taken 
subsequent to the training. With one notable exception, there were no significant 
differences found among the post-training, two, sbc and 12-month surveys. That 
exception is the assertiveness scale, “Willingness to Voice Disagreement,” which increased 
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significantly between the post and 2-month surveys. All four attitude scale scores 
remained high thereafter over the remaining period. 


Change in the Control Variables for the Pre-Post and Follow-up surveys. The two 
"Goal Sharing" measures were also compared over the same time period as the four 
attitude scales. The goal sharing scales were not expected to change as a result of the 
CRM training. Although goal setting and attainment are important aspects of maintenance 
work, the training was not designed to directly influence either attitudes or perceptions 
about that aspect of management. Figure 2 presents the pre, post, 2, 6, and 12 month 
comparisons for the two goal sharing scales using all responses. The MANOVA “F’ test 
was applied to these data and was found not to be statistically significant (F=1.29, 
p>.05,df= 2156,4). 


FIGURE 2 


Goal Attainment (Control Scales) 


j □ Pre-Training (n=2056) | 

I B Post-T raining (n=2053) i 
I □ 2-Mo Follow-op (n=81 0) | 
I □ 6-Mo Follow-up (n=722) ■ 
: B 1 2-Mo Follow-up (n=540) | 

Goal Goal 

Sharing in Sharing w/ 

Own Group Other Gps. 



Matching Questionnaire Responses Over Time 

Matched questionnaires and sample size. For the remainder of the analyses in this 
report the results for the CRM/TOQ will be reported in terms of the largest number of 
individual surveys received after training that could be matched, by I.D. code, to their 
pre-training counterpart. 

CRM training participants were asked to include a confidential identification 
number on the top of their pre-training questionnaire and to write that number on the firont 
page of their training notebook. They were encouraged to invent a sbc-digit numb^ 
known only to them to ensure complete confidentiality. The participants were alerted that 
they would be asked to use the number again on a similar questionnaire at the completion 
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of the training, as well as three more questionnaires which would be mailed to them in the 
months to follow. This I.D. number, it was explained, would be used over the times 
sampled to compare answers respondent by respondent rather than merely by groups over 
the four time periods. Participants were assured that the numbers they chose were not 
intended to identify or single out individuals, but merely to match earlier answers on the 
questionnaires with later ones. They were also told that this questionnaire, and all others 
to follow, would be sent directly to the University for processing, tabulating, and 
summarization. 

Rationale for the “matched respondents” sample. Using a matched sample insures 
the comparisons made over time are made with the same core group of people, which 
provides added validity to the time-lagged results obtained. One practical cost of this 
matching is that the size of the matched sample for each of the three follow-up surveys is 
reduced, over the total, by the number of respondents who did not include a recognizable 
code with their completed questionnaire. The benefit of using such a sample, however 
clearly outweighs the penalty of a reduced sample size. Using matched respondents 
overtime guarantees that a work group’s average attitude scores, compared over time, will 
be drawn fi'om the same sample. There need be no concern that differences found over 
time are the result of comparing respondents who are incomparable with one another. 

Table 4 displays the number (and percentage) of respondents whose later answers 
could be matched with their answers to the previous questionnaires. For the analyses 
included in this report the sample used is of respondents whose subsequent answers were 
matched to their pre-training responses. Table 4 shows that sample, matched to their pre- 
training responses (shown in the upper row in Table 4), to include between 74% and 88% 
of the total responses available. 


TABLE 4 


Numbers of Questionnaires Matched with Previous Surveys 



Post-training 
Survey (n=2,053) 

2 Month Follow- 
up Survey 
(n=810) 

6 Month Follow- 
up Survey 
(n=722) 

12 Month 
Follow-up 
Survey (n=540) 

Matched with 
the Pre-training 
survey (n=2,056) 

1 ,822 (88%) 

598 (74%) 

548 (76%) 

415 (77%) 

Matched with 
Post-training 
Survey (n=2,053) 

- 

561 (69%) 

520 (72%) 

387 (72%) 


Testing for differences among the matched samples, over time. The demographic 
characteristics for the post-training sample, and the three follow-up samples were 
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compared with one another, in order to test for possible differences in the composition of 
the four groups. Table 5 presents those comparisons for age, occupation, department, 
education and prior experience. 


TABLES 

Demographic Comparisons Among the Four AftervTraining Surveys 


Demographic 

I 

Posting. 

2 Month 

6 Month 

12 Month 

Statistical 

Significance 

Item 

Survey 

Survey 

Survey 

Survey 

Test 

Level 

Job Title (9) 

Engineers 

6.6% 

8.8% 

9.3% 

12.4% 

i’=82.4 

.000 

Inspectors 

9.6% 

3.8% 

3.5% 

1.8% 

df=24 


Oeoartment |6| 

Engineering 

7.1% 

9.3% 

10.8% 

13.1% 

l’=51.0 ■ 

.000 

Qual. Cant 

11.3% 

7.3% 

6.7% 

6.6% 

df=15 


Location (City 

n/a 

n/a 

n/a 

n/a 

x’=28.6 

ns 

» 14) 





dt=39 


Mean Age in 

42.0 

43.0 

42.5 

43.9 

■r => 3.73 

.001 

years 





df=3, 3220 


Education 

Mean years ] 

5.7 

5.2 

6.3 

I 

5.7 

•r = .29 

ns 

College 





df=3. 3196 


Mean years 

5.1 

4.4 

5.5 

4.9 

•r =.32 

ns 

Trade Sch. 





df=3, 3242 


Military Exp., 

6.1 

5.7 

6.5 

6.3 

CO 

II 

tl. 

ns 

Mean Years 





df=3. 3327 i 



Age, years of education, and years experience were tested using the ANOVA “F,” while 
the differences among the four survey samples on occupation, and department (being 
categorical or “nominal scale” data) were examined using the “chi-square” (zO test. Of all 
the tests performed for these data, only age, occupation and department were found to 
show significant differences among the four time samples. Not unexpectedly, the average 
age of the matched sample increases firom 42 years at the time of training, to almost 44 
years at the last follow-up survey. Because many 12 months surveys were sent out late 
and delays were seen in their return, this increased age doubtless reflects the mere passage 
of time. Closer examination of the results for occupation revealed that of all the 
occupations a disproportionately large number of Engineers, and small number of Aircraft 
Inspectors, are found in the 12 month sample when compared with those occupations' in 
the post-training survey. Likewise, for the test of diSerences among departments show 
that Engineering Department and Quality Control (the Inspectors' department) to show 
disproportionately higher and lower numbers, respectively, at 12 months following their 
training. Given that Engineers and Inspectors, togetha; account for only 15-17% of the 
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total sample of matched respondents, the effect of their over and under-sampling at 12 
months is unlikely to affect the overall results reported below. 

Number of work units represented in the “matched sample.” The total of all 1,822 
matched respondents in the post-test sample, includes the members of over 50 
organizational units drawn from all parts of Technical Operations; (Supply, Quality, Base 
Maintenance, Line Maintenance, Shops, Stores, Materials, Engineering, Planning, and 
Administration). The total numbers of units in each analysis vary depending on the 
specific performance indicator, because not all the work units are measured on the same 
performance. These differences are described in the following section. 


RESULTS SECTION 2; TRENDS IN MAINTENANCE PERFORMANCE 
BEFORE AND AFTER THE ONSET OF CRM TRAINING 


Performance Data Described 

Maintenance managers in the company collected performance data in abundance. 
Table 6 presents the five measures used as end-result criteria in the present study. 


TABLE 6 

Performarice Measures Selected for Test 


PERFORMANCE 

MAXIMUM WORK UNITS 

RATED SENSITIVITY 

MEASURES 

MEASURED 

(1 = most sensitiye to tmg) 

SAFETY 

Aircraft Safety 
(number of maintenance- 
related ground damage 
incidents per flight) 

38 

1 

Occuoational Safety 

(lost time injuries, per 1 000 

hours) 

55 

1 


DEPEND A BIL I TY 


bn-time Deoartures 
(% departures w/in 5 minutes 
of schedule) 

31 

1 

On-time Maintenance 
(delays due to late from 
planned maintenance, per 100 
scheduled aircraft) 

35 

1 


EFFIC IEN CY 


Contained Oyertime Cost 

55 

2 

(% oyertime paid to total wage i 

-bill} L_ 
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Three conditions were met in order to include these measures in the work-unit analysis 
reported here. First (and obviously) the performance measures need to be available by 
work unit (i.e., line station, shop, warehouse, etc.), and not just by department or function. 
Second the measures must be ones that people in the work unit can afifect by their actions, 
or can effectively adapt to; and not merely ones that are conveniently assigned to a unit, 
but for which little can be done. The third condition applied was that the measures not be 
directly related to (or completely determined by) other measures in the set. The five 
performance measures used in the present analysis are included in the company's 
categories of "safety," "dependability," and "efficiency." "Quality" measures used by the 
company are available only for the whole department. Because they cannot be applied to 
the specific maintenance units, quality measure are not included in this present analysis. 

The trainers and administrators of the CRM course evaluated all the available 
performance measures and predicted which of them would be more sensitive to effects of 
CRM training. Their conclusions were that four measures would be the most readily 
improved by the training. These four performance measures included the two safety items 
“ground damage” and “days lost to occupational injury;” and two dependability measures, 
“flight departures within 5 minutes,” and “delays in ‘scheduled work’ (e.g., overnight 
repairs, and overhaul) due to maintenance error”. Their lower ranked performance 
measures in responsiveness to training included the efficiency measure of “overtime paid.” 


Performance Rates and Trends Before and After the CRM Training Began. 

The "before" and "after" scores for the five performance measures were plotted, and their 
trends compared over the time period for which measures were available. 

Measures of Safety Performance: Safety of Aircraft and Personal Safety. Figure 3 
presents before-and after-training comparisons of number of Ground-Damage incidents for 38 
work units. 




Figure 3 shows a strengthening of the trend for the decreased number of ground 
damage incidents (measured in incidents per flight), for the 30 months (for which there were 
data available) after the CRM training began, when compared with the six months before. The 
reduced rate for ground damage incidents appears to be relat^ in time to the CRM 
training. 
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Figure 4 shows the rate of lost time injuries, per 1000 hours worked, for 55 work 
units.. In Figure 4 we see a virtual reversal for the after training trend for occupational injury for 
the 33 months after the onset of training, compared with the six months before. 

Measures of Dependability. Figure 5 presents before- and after-training comparisons 
for “Departures within 5 minutes of schedule” for the 3 1 line maintenance stations for which 
this measure is relevant. 


FIGURE 5 


Dependability: (On-time Departures) 



♦ Depend/Pre Bl Oepend/Post - - “ PreTraining Trend ' PosITraintng Trend | 


Figure 5 demonstrates that line maintenaiKe responsibility for on-time departures 
continues to steadily improve both before and after the training began. Although the 
performance was trending up in the period preceding the training, this chart shows a 
higher level of performance, overall, at the end of the second year following the onset of 
CRM training. 

Figure 6 shows before and after comparisons for the other indicator of dependability, 
“On-time planned maintenance” — measured by the number of aircraft delays due to late from 
scheduled (or planned) maintenance - for 35 work units (including line and heavy 
maintenance stations). 
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FIGURE 6 


Dependability: Late from Planned Maintenance 



...... Delays, Pre-training — 0 — Delays, Post-training 

" - • Post-training Trend ~ Post-training Trend 


Although the overall rate for "On-time maintenance” (i.e., delays due to being late 
from scheduled maintenance declining) continually improved over the period measured, no 
appreciable change in that rate is noted in the years following the onset of CRA'I training, 
when compared with the period before the training began. 
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Measure of EflSciencv. Figure 7 shows the pre-post comparisons for the percentage of 
hours of overtime charged for the 55 work units measured. 



Figure 7 shows that the trend of overtime expenses for the 16 months following the onset 
of CRM training continues the rate of improvement, but that trend then reverses from 
December, 1992 for the remaining 16 months for which we have data. The initial 
improved cost performance appears coincident with the training, and may be affected by it. 
An alternative explanation is that these rates were more affected by top management 
policy in 1991 and 1992, which at first sustained tight controls on overtime expenses, and 
then undertook a labor force reduction which required increased overtime for the 
remaining staff. Further analyses below examine this possible effect of changed policy and 
of CRM-related skills used in coping with it. 
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RESULTS SECTION 3: TESTING RELATIONSHIPS 
BETWEEN ATTITUDES AND PERFORMANCE 

The data sets used in the attitude-performance analyses. There were five data sets 
we used for the attitude-performance analyses. By the completion of the current phase of 
the CRM training, in March, 1994, 1,822 of the 2,199 participants attending one of the 
CRM workshops had completed questionnaires such that they could be included in the 
“matched samples” of the pre-, and post-training surveys as well as the 2, 6, and 12 month 
follow-up surveys. Of those 1,822, 1,692 participants had also provided valid 
departmental identification and, thus, could be included in one of the five data sets. The 
first of those data sets was comprized of 395 participants who were onployed in the 
Technical Operations department’s Line Maintenance division. Attitudes fit>m those 395 
Line Maintenance participants in 3 1 work units are tested against the Line Maintenance 
measure of “On-time departures” For the second data set. Base Maintenance division 
added another 266 participants (in 4 hangar locations), which together with Line 
Maintenance participants provides a total of 661 training participants (in 35 work units) 
for the source of attitude data examined in relationship with the maintenance measure 
“On-time maintenance.” The third data set added another 477 (in three more city-site 
locations) from the Technical Operations department’s direct support divisions, of Quality 
Control (n=192), Prime Shops (n=l 1 1), and Materials Service (n=174), to test the 
performance measure “Aircraft safety. ” That third data set is composed of 1, 138 
maintenance and support personnel in 38 work units. For the fourth and fifth data sets, 
participants in the remaining Technical Operations units for Planning (n= 194), Engineering 
(n=120), and Administration (n=240) were added to the data sets above for a total of 
sample of 1,692 participants in 55 work units. These samples were used to test attitudes 
against the performance measures of “Ocaipational safety “uid “Overtime costs 
contained. ” 

The correlation statistics used in the present analysis. The relationships between 
the attitude indices and the performance measures, as presented in Tables 7-1 1 below, 
were calculated using the Spearman Rank-order Correlation statistic ("Rho," or p). The 
use of Rho is advisable in this case because the distributions of these five performance 
measures narrow the choices to t non-parametric statistical tests for optimal analytic 
power (Taylor, 1991b). The measured results of four of the five performance measures 
are “improving” when they decline numerically (i.e., the absence of delays due to late fi'om 
planned maintenance, the absence of occupational injuries, the absence of ground damage 
incidents, and reduction of overtime costs). To simplify the presentation of findings all 
results in Tables 7-11 are described as postive coefficients when the original correlations 
are in the expected direction (i.e., favorable attitudes equal better performance). 
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Lead and Lag Effects Between Performance and Attitudes Over Time 

A part of the following analysis examines the leading impact of performance prior 
to the CRM training on participants’ subsequent attitudes. The more central part of the 
evaluation of CRM training however, is the analysis of those attitudes' lagged effects on 
subsequent maintenance performance. We are interested mainly in the training’s longer- 
term effects on attitudes, and their lagged effect on performance following the training. 

The relationships between the pre-training questionnaire scales and maintenance 
unit performance measures. The first associations examined here are the four pre-training 
attitude scales (measured at the beriming of each training session) with the five 
maintenance performance indicators. Table 7 contains the percentages for statistically 
significant correlations between each of the four attitudes and each of the five performance 
indicators. The larger percentages, of 1/3 (33%) or greater, are boldfeced as a visual aid. 


TABLE 7 

Percent of Significant Correlations 
Between Pre>training: Attitudes and Perfonnancs::: 


Pn-Trafning Attftudo Scaloa 


Sharing Command 

Communication & 

Managing Stress 

Assertive- 

Responsibility 

Coordination 

Effects 

ness 


Dependability 


Departures within 5 min of schedule 


1 3 months before traininq ] 

40 


40 

33 

1 20 

26 months after training | 

33 ; 


40 

11 

[ is 


On-time maintenance 


17 months before traininq 

6 

39 

0 

6 



0 

40 

7 

15 


Safety 


Occupationai Safety 


6 months before traininq 

40 I 

40 I 20 

0 

33 months after training 

6 I 

S8 I 3 

24 


Aircraft Safety 


6 months before traininq 

0 

40 


0 

20 

28 months after training | 

6 

13 


10 

3 


Efficiency 


Overtime paid 


6 months before traininq 

1 .0 ..] 

20 

1 P 

0 

33 months after training 

1 .« _.i 

11 

1 .1.5 

21 


Table 7 shows that the pre-training surv^ attitude scale “Communication & 
Coordination” has a large number of statistically significant relationships with four of the 
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five pre-training performance indicators. That means that good performance before the 
CRM training began was quite widely related to positive pre-training attitudes about 
communication & coordination. That pre-training scale measuring communication & 
coordination attitudes was widely related to post-training dependability and occupational 
safety as well. Taken together, these results suggest that although communication and 
coordination is an important value to managers and professionals in the most effective 
units, CRM training may not be a crucial intervention for this effect. This communication 
and coordination scale had a remarkably high initial mean score {cf.. Figure 1), which 
suggests the possibility that the scale may contain a “social desirability” bias. 
“Communication and cooperation” is also quite a general concept and can have vague or 
multiple meanings. It is also a somewhat “passive concept.” Because its constituent 
questions have fewer specific behavioral references than the other more specific attitudes 
measured with the CRM/TOQ. 


FIGURE 8 


Correlations between: Pre-Training Attitude “Communication 
& Coordination,” and Selected Performance Indicators 


Training 

JSegirt* 


c 3 

.2 c 

S V 


O O 



a Aircraft Safety (n=35) ■ Occupational Safety (n=50) □ On Time MX (n=35) □ On Time Departures (n=30) 


Figure 8 plots all of the individual correlations between “communication & 
cooperation and the four performance indicators most related to it. Figure 8 reveals a 
large number of months in which the correlations between that attitude and the four 
performance measures are as high as .40. Over time some of those correlations, 
particularly for aircraft safety and on-time maintenance, decline in size. To illustrate this 
Table 7 shows “communication & coordination” related to “aircraft safety” 40% of the 
months measured before training began in June, 1991. In Figure 8, aircraft safety is shown 
as the first profile of the four. That profile of relationships between and communication & 


27 





( 


coordination with aircraft safety demonstrates than the correlations are higher and 
proportionately more numerous before the training began, than afterwards. 


Table 7 also shows that another pre-training attitude with many significant 
correlations with performance is “Sharing Command Responsibility.” This attitude scale is 
seen to be frequently related to line station dependability (“departures within 5 minutes of 
schedule”) both before and after training began. Pre-training “sharing responsibility.” is 
also related to containing subsequent overtime costs. Figure 9 displays the all the 
correlations between the pre-training attitude toward sharing responsibility, and overtime 
costs. 


FIGURE 9 


Correlations between: PreTraining Attitude about Delegation, 
and Overtime Costs Contained 



Figure 9 is a more simple two-dimensional presentation than Figure 8. Figure 9 
allows for the inclusion of two additional features. The first feature is a solid line showing 
the level of significance (p =,22, p< 5%) for this performance measure using the 55 work 
units available for this series. The second feature is a dotted trendline (calculated bythe 
Least-Squares Regression method) for the correlation coefficients over time. Figure 9 
provides the information from Table 7 in graphical Form. It shows that of the 33 months 
of overtime measured after training, 14 of them (42%) had correlations above the 
significance level. Furthermore, the trendline in Figure 9 shows an upward slope which 
confirms that the correlations between pre-training feelings about participation (“sharing 
responsibility”) and subsequent coping with restrictive overtime policy become more 
strongly related over the nearly three year period, 6/91-3/94. The results in Figure 9 
clearly show that managers and staff professionals who hold more participative values 
before they attended training are subsequently successful in coping with overtime costs. 
Whether the CRM training can impact to further increase that value will be tested below. 



The relationships between the post-training questionnaire scales and maintenance 
unit performance measures . Table 8 displays the percent of significant correlations 
between attitudes obtained from the post-training survey and the five performance items. 


TABLES 

Percent of Significant Correlations 
Betweeni Post-training: Attitudes and Peif^ 


Post-Training Attitude Seaies 




Communication t 

Managing Stress 

Assertive- 



Coordination 

Effects 

ness 


Dependability 


Departures within 6 min of schedule 


1 3 montPs before training I 

0 

58 

17 

17 

26 months after training | 

7 

29 

17 

4 


On-time maintenance 


1 7 months before training 

6 

53 

41 

24 

27 months after training 

7 

33 

37 

22 


Safety 


Occupational Safety 


6 months before training 

20 1 

20 

0 

0 

33 months after training 

? J 

3 

3 

13 


Aircraft Safety 


6 months before training 

40 

0 

20 I 

0 

28 nwths aftw traim^ 

3 

9 

3 1 

3 


Efficiency 


Overtime paid 


6 months before training 

40 

0 

0 

0 

33 months after training 

is 

3 

12 

3 


Improved attitude about “sharing command responsibility” is seen in Table 8 to be 
related to preceding months of good aircraft safety (lower damage) and lower overtime 
costs. These results suggest that the CRM training’s immediate positive effect on 
attitudes about participation cannot be credited with subsequent performance. Table 8 
also shows that the post-training attitudes about communication & coordination, and the 
“value of stress management,” are both very frequently related to on-time scheduled 
maintenance performance both prior to and after the training began. As noted above, the 
attitude scale measuring “communication & coordination,” may be subject to some “social 
desirability” bias, but it is apparently also sensitive to the effects of the training. Because 
the attitude about stress management is lower overall before the CRM training, and the 
training specifically targeted improving stress management, it seems even more likely that 
the training has a direct effect on that attitude and can be credited with subsequent 
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performance related to it. Thus, the number of significant correlations between stress 
management and performance for the months after the training began, are likely to be the 
result of CRM training. 

Figure 10 displays all of the correlations between on-time scheduled maintenance 
performance and the two post training attitudes, “communication and coordination,” and 
“stress managment.” Figure 10 reveals that the pattern for correlations between the two 
attitudes, and pre and post training performance, is similar although communication & 
coordination has higher correlations before the onset of training, with on-schedule 
maintenance, than does stress management. 
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The relationships between the follow-uo surveys and maintenance unit 
performance measures. Tables 9, 10 and 1 1 display the percent of significant correlations 
between attitudes obtained fi-om the 2-month, 6-month, and 12-month follow-up surveys, 
respectively, and the five performance items. 


TABLE 9 

Percent of Significant Correlations 
Between 2^onth Fo)Iow-up Attitudes and Performance 


2 Month AWtutia Seales 




CommunieatiM 9, 

Managing Stress 

Assertive- 


m^SSSiSI^KK^ 

Coordination 

Elteets 

ness 


Dependability 


Departures within 5 min of schedule 


1 3 months before traininq 

75 

8 

17 

33 


52 

11 

7 

52 


Qn4ift>e maiirtenance 



65 

0 

0 

0 

27 months after training | 

78 

4 


11 


Safety 


Occupational Safety 


6 months before training 

20 

20 

0 

100 

33 months after training 

32 

3 

9 

62 

Aircraft Safety 

- 

6 months before traininq 

40 

20 

20 

20 

28 months after training 

29 

4 

14 

21 


Efficiency 


Overtime paid 


6 months before traininq 

0 

20 

20 

20 

33 months after training 

9 

3 

21 

3 


Table 9 displays the proportion of significant correlations between maintenance 
unit performance and attitudes from the follow-up survey obtained 2-months after training 
for those respondents with questionnaires matched to their pre-training survey. The more 
passive belief about “communication & coordination” was not found related to subsequent 
maintenance performance in this or in any later survey. Examination of these 2-month 
follow up results, reveals stronger relationships with performance, using attitudes about 
“sharing command responsibility,” and the value of “assertiveness,” than the immediate 
post-training survey showed. Both “sharing command responsibility,” and “assertiveness,” 
are "active" beliefs, the behavioral skills for which were emphasized in the training; and 
which require continued behavioral support, following the training, to reinforce them. 
Sharing responsibility, measured 2 months after training, shows a great many months of 
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significant correlations to post training performance for all dependability and safety 
indicators. 

Figure 1 1 displays all of these correlations for the “command delegation” scale. 
Note how high many of the correlations are ~ particularly with the two dependability 
scales (on-time departures, and on-schedule maintenance). 


FIGURE 11 


Correlations between: 2 Mo Attitude, “Sharing Cmd. Responsibility," 
& Selected Performance Indicators 
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B Aircraft Safety (n=35) □ Occupational Safety (n=50) B On-time Departures (n=30) ■ On time Mx (n=32) 


Table 9 also shows that assertiveness is related to many (mostly subsequent) 
months of on-time departures, and occupational safety. Figure 12 displays these 
correlations between “assertiveness” two months after training, and performance. 
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FIGURE 12 


Correlations between: 2 Mo Assertiveness Attitude, 
& Selected Performance Indicators 



□ Occupational Safety (n=50) □ On-time Departures (n=30) 


Overall, the results from this two-month follow-up survey are remarkably strong 
and suggest that lessons learned from the CRM training have consolidated by two months 
afterward, and in more "active" ways. 

Table 10 displays the percentage of significant correlations between the attitudes 
measured six months after training and all available months of performance results. 
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. : Betwee n iS-M^ Fp llpyf^pr^^ ■ 


S Mofith Attttude Scatas 




Coamwnleatlon t 

Managing Str^s 

Assardva- 



Coordination 

Enacts 

ness 


Dependability 


Departures within 5 min of schedule 


13 months before traintnq 

16 

42 

0 

0 

26 morths after^tralrHng 

4 

26 

0 




On-time maitrtenance 


17 months before training 

18 

18 

0 

6 

27 months after training 

19 

0 

4 

4 


Safety 


Occupatioiul Safety 


6 months before training 

40 

1 ^ 

40 

0 

33 months after training 

21 

1 is 

21 

3 


Aircraft Safety 


6 months before training 

20 

40 

0 

0 

28 months after training 

9 

8 

6 

P 


Efficiency 


Overtime paid 


6 months before training 

0 


20 


20 

20 

33 months after training 

9 


3 


21 

3 


These 6 months survey results show a lapse in the effects of CRM training compared with 
the 2-month follow-up survey. They reveal that only prior maintenance performance is 
related to opinions about delegation, communication, and stress management, surveyed six 
months after training. Pretraining occupational safety, was the only performance indicator 
widely correlated to three of the four attitude scales measured six months after training. 
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Figure 13 displays all the correlations for occupational safety and the three 
attitudes. Although many of the correlations in Table 10 are statistically significant, most 
of them are lower in size than those seen for the two month follow-up survey in Figures 
11 and 12. 


FIGURE 13 


Correlations between: Occupational Safety, & Selected 6 Mo 

Attitudes 



□ Stress Management ■ Communtcation & Coordination S Command Delegation 


By the 12-month follow-up survey, the training effects have returned to mirror 
aspects of the post and 2-month surveys. Table 1 1 presents the percentage of significant 
correlations between the five performance measures and those attitudes. 
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TABLE 11 

Percent of significant Correlations; 

; / Between iZrlVlbritfiFolIpW-^pfAttitud^^^ ":v ^ 


12 Month AWtuda Scales 




Communication & 

Managing Stress 

Assertive- 



Coordination 

Effects 

ness 


Dependability 


Departtires within S min of schedule 


30 months before survey | 

25 

0 

13 

0 

1 4 months after survey | 

■ 7 

0 

13 

7 


On-time maintenance 



6 

6 

23 

0 


11 

0 

52 

0 


Safety 


Occupational Safety 


1 8 months before survey 

35 

12 

18 


78 

21 months after survey 

9 

5 

14 


68 


Aircraft Safety 


1 8 months before survey 

6 

6 

12 

0 

1 6 months after survey 

0 

5 

P 

5 


Efficiency 


Overtime paid 


1 8 months before survey 

24 1 

5 

29 

18 

21 months after survey 

27 ! 

27 

SO 

5 


According to Table 1 1, attitudes about the value of assertiveness one year after CRM 
training were related to a great many months of occupational safety both before and after 
that survey was taken. Attitudes, a year after training, toward sharing command decisions 
were (reminiscent of the 2 month results) also related to many months of occupational 
safety, but this time, in the months before the 12-month survey. Similar to the 2 month 
results, assertiveness attitudes were also related to many months of occupational safety 
performance. Figure 14 shows the correlations between occupational safety and the two 
12 month attitudes “sharing command” and “assertiveness.” 
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FIGURE 14 


Correlations between: Occupational Safety, 
and Selected 12 Mo Attitudes 



a Command Delegation 12 months after training QAssertivefiess 12 months after training 


When compared with the decline of subsequent training results from the sbc month 
follow-up outcomes these continued improvement of relationships with attitudes after 12 
months reflects further strengthening of the training effects. This suggests that the 
observed diminution of associations using the earlier, sbc month attitude data {cf.. Figure 
13)“ conceals an internal consolidation of training effects rather than their alleviation. In 
effect, maintenance managers and staff professionals recognized, in the months following 
CRM training, that their new knowledge and attitudes about participation was of value 
and that communication, and the effect of practicing the appropriate behaviors, could have 
an effect on their units’ safety. 

Another, very interesting outcome of the 12-month survey, shown in Table 1 1, 
were the relationships between managing stress effects and the many subsequent months 
of on-schedule maintenance, and overtime cost savings. These results follow June 1992, a 
full year after the onset of CRM training in June 1991. They represent performance for 
both of these very stressful requirements (holding to planned maintenance schedules and 
staying within required overtime levels) for an additional lagged period during which the 
lessons of stress management, learned during training a year earlier, had also consolidated 
for the training participants. Figure 1 5 presents the correlations between those stressful 
requirements and 12 month attitudes toward stress management. 
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RESULTS SECTION 4: 

MANAGERS SELF-REPORTS OF BEHAVIOR CHANGE: 
ANALYSIS OF RESULTS OF FIELD INTERVIEWS, AND 
WRITTEN COMMENTS FROM THE POST-TRAINING 
AND FOLLOW-UP CRM/TOQ 

Two open-ended questions were asked of the maintenance managers concerning 
the intended use of the CRM training and its application to the job. What follows are the 
results of these two questions. The fiill sample available by October, 1994 is employed in 
the following analyses. 
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Intended use of CRM trainina. Figure 16 shows that respondents steadily 
increased their reported intention to do what we classified as actively “dealing with 
others” over the four surveys taken after training. They correspondingly reduced their 
reported intention to passively “be a better listener.” 
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Reported use of CRM training. Figure 17 shows that, after two months, 
respondents reported less passive (‘better listener”) application of the training. By 12 
months after their training these respondents reported activities we could classify as 
“dealing better with others.” Thus, in the months following training, increasing use of 
active skills form the training was intended and achieved. 


FIGURE 17 


Reported Use of the Training 
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x’=14.3, df=4, p<.01 


CONCLUSIONS 


The increased safety, dependability and improved costs associated with positive 
attitudes following the CRM training are strong effects, and they are attributable to that 
training. The impressive number and size of the correlations between post training 
attitudes and performance, together with the longitudinal association with performance 
months after the training are very convincing evidence for the positive effects of CRM 
training. 
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Comparisons of managers' attitudes immediately after their training with their pre- 
training attitudes show significant improvement. Improvement took place in attitudes 
about "willingness to share command responsibility," "usefulness of communication & 
coordination," and "recognition that stressors aSect decision making." The same attitudes 
of these managers two-, six-, and 12-months later reveal that these favorable post-training 
attitudes remain at those positive levels in the months after the training. The fourth 
attitude scale measured, “willingness to voice disagreement” (a measure of assertiveness), 
although showing no significant change immediately following training, improved 
significantly above the pre-training levels two months after training and it remained at that 
higher level six and 12 months afterwards. The influence of the training on all the 
expected participant attitudes is thus a stable and robust change and not merely a brief 
"honeymoon” effect. 

Other analyses of the data looked at the relationships between the four attitudes 
and maintenance results over several years. In this “time-lagged” or longitudinal 
correlation analysis we tested the effects of the training program on performance. We 
compared performance before the onset of the training, to attitudes in subsequent time 
periods, as well as comparing attitudes with later performance results. We found an 
explainable pattern of significant relationships between the participant’ pre-training and 
post-training attitudes, and between those attitudes and performance. Positive attitudes 
about “assertiveness,” “sharing responsibility,” and “stress management” — all skills and 
behaviors specifically taught in the CRM training — were more predictive of subsequent 
performance. Positive attitudes about the more passive belief^ “communication and 
cooperation,” seemed less sensitive to the CRM training and were less predictive of 
subsequent performance. That attitude, communication & coordination, was not found 
related to maintenance performance in any of the follow-up surveys. Positive attitudes 
were apparently initially valuable, as they suggested a sensitivity to more specific 
behaviors and later attitudes about assertiveness, stress management, and delegation. 

Associations between positive post-training attitudes and subsequent performance 
were even stronger when attitudes were measured at 2 months and at 12 months after the 
training. Overall, the survey results firom these 2 month and 12 month follow-up surveys 
are remarkably strong and indicate that lessons learned from the CRM training consolidate 
and strengthen in the months afterward, and in more “active” or practical ways. In most 
cases the strongest relationships were found in the months following the measurement and 
they then trended downward for the remaining period. 

The strong implication is that CRM training for maintenance managers and support 
professionals is effective if the behaviors taught are given time to develop. Once 
competence and early success is achieved consolidation takes hold and reinforces the 
positive attitudes making them more enduring. Simply letting time take its course does 
not assure the continued success of the program. The period between two months and 
one year following initial training should be an ideal time for recurrent training to sharpen 
the skills and to strengthen management resolve. Additionally the motivation and logic to 
extend the CRM training to mechanics is strong among past participants and this difiusion 
should be realized. With diffusion to mechanics, and recurrent training for all participants. 
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we should then expect to find that ever increasing maintenance performance would be the 
outcome of the positive attitudes and inspiration resulting fi'om “crew resource 
management.” 
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APPENDIX B 


Date 


) 
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TECHNICAL OPERATIONS DIVISION 
CCC WORKSHOP SURVEY 


“Six-inonth Follow-up'* Questionnaire 


Please enter the five digit Personal Identification Number 
that you selected at the beginning of the seminar. 

Identification Code 


Now, please answer by writing beside each item the niimber that best 

reflects your personal attitude. Choose the number from the scale 

below. All data are strictly confidential. 

*****gcale***** 

1 2 3 4 5 

Disagree Disagree Neutral Agree Agree 

Strongly Slightly Slightly strongly 

L . Technical Operations team members should avoid 
disagreeing with others. 

2. It is important to avoid negative comments about the 

procedures emd techniques of other team members . 

3. Casual, social conversation on the job during periods of 

low workload can improve Technical Operations team 
coordination. 

4. Good communications and team coordination are as 

important as technical proficiency for aircraft safety 
and operational effectiveness. 

5. We should be aware of and sensitive to the personal 

problems of other Technical Operations team members. 

6. The mcuiager, supervisor, or assistant supervisor in 

charge should take hands-on-control and make all 
decisions in emergency and non-st 2 mdard situations. 

7. The manager, supervisor, or assistant supervisor in 

charge should verbalize plems for procedxires or actions 
and should be sure that the information is understood 
and acknowledged by the other Technical Operations 
teeun members. 
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*****Scale***** 

1 2 3 4 5 

Disagree Disagree Neutral Agree Agree 

Strongly Slightly Slightly Strongly 

_8 . Technical Operations team members should not question the 
decisions or actions of the manager, supervisor, or, 
assistant supervisor except when they threaten the safety 
of the operation. 

_9. Even when fatigued, I perform effectively during critical 
phases of work. 

10. M 2 magers, supervisors, and assistant supervisors should 
encourage questions during normal operations and in 
special situations. 

11. There are no circxmstances where the subordinate should 
assume control of a project. 

.12. A debriefing <uid critique of procediires emd decisions 

after each major task is an important part of developing 
and maintaining effective team coordination. 

.13. Overall, successful Technical Operations mamagement is 
primarily a function of the manager's, supervisor's, or 
assistant supervisor's technical proficiency. 

14. Training is one of the manager's most important 
responsibilities . 

.15. Because individuals function less effectively under high 
stress , good team coordination is more important in 
emergency or abnormal situations. 

.16. The start -of -shift team briefing is important for safety 
amd for effective team mamagement. 

_17. Effective team coordination requires each person to take 
into account the personalities of other team members. 

_18. The responsibilities of the manager, supervisor, or 

assistant supei-visor include coordination between his or 
her work team and other support aureas. 

.19. A truly professional manager, supervisor, or assistant 
supervisor can leave personal problems behind. 

.20. My decision-making ability is as good in aUsnormal 
situations as in routine daily operations. 
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**** * 50318 ***** 


1 

Disagree 

Stirongly 


2 

Disagree 

Slightly 


3 

Neutral 


4 

Agree 

Slightly 


5 

Agree 

Strongly 


In the following questions, ”my manageaent group" refers to those 
people who report to the same manager that I do. 

21. I am kept informed by others in my management group about 

the goals and objectives of this organization (e.g., 
cost, quality, service, etc.). 

22. Work goals and priorities are understood and agreed to by 

members of my management group. 


In the following items, "my work group" refers to those people who 
report to me. 


23. Employees in my work group receive detailed feedback 
regeirding the orgamization's performamce. 

24. If employees in my work group disagree with the goals and 
priorities that have been established, they feel free to 
raise their concerns with supervision. 

25. Employees in other groups within Technical Operations plan 
and coordinate their activities effectively together with 
people in my work group. 

26. Employees in other groups, departments and divisions 
throughout the company act as if they share mainy of the 
same organizational goals that we do. 


27. How useful has the CCC training been for others? (circle 
one) 

A Waste Slightly Somewhat Very Extremely 

of Time Useful Useful Useful Useful 

28. How much has the CCC training changed your behavior on 
the job? (circle one) 

No Change A Slight A Moderate A Large 

Change Change Cheinge 
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29. What cheuiges have you Dade as a result of the CCC 

training? 



30. How will you fuller use the. CCC training in the coming 
months? 





Year of birth 


Total years at 
Sex (M or F) 



This conpletes the Questionnaire 
Thcuiks for your help. 
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